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Objective 


The  goal  of  the  research  was  to  create  enhanced  practical  methods  to  estimate  and  simulate  hyper¬ 
spectral  images  by  using  recorded  data  of  low  spectral  resolution  images.  These  data  can  be  used 
to  test  algorithms  designed  for  many  hyperspectral  image  analysis  tasks,  such  as  target  detection, 
classification  and  tracking.  An  advantage  is  that  these  images  are  unclassified  and  can  be  used  in 
a  university  environment  where  international  students  are  often  used.  The  images  are  also  readily 
defined  to  allow  ground  truth  to  be  known  so  that  accuracy  of  the  image  analysis  tasks  can  be 
measured. 


1  Introduction 


This  work  developed  and  implemented  basic  theory  for  estimation  and  simulation  of  realistic  hy¬ 
perspectral  images.  The  dual  goals  of  both  estimation  and  simulation  are  reasonable  since  the 
proposed  process  for  creating  simulated  hyperspectral  images  begins  with  a  multiband  image  of 
much  lower  dimensionality,  e.g.,  R,G,B.  Since  this  image  provides  a  starting  point  for  the  creation 
of  the  hyperspectral  image,  we  can  consider  the  simulated  image  an  estimate  of  an  actual  hyper¬ 
spectral  image.  The  quality  of  the  simulations  can  be  judged  not  only  by  how  realistic  they  appear, 
but  how  well  they  match  a  true  hyperpsectral  image  that  is  used  to  create  the  lower  dimensional 
data. 

This  effort  resulted  in  methods  to  create  hyperspectral  images  based  on  common  three  band  (RGB) 
images.  The  images  that  were  created  appeared  indistinguishable  to  the  standard  observer  in  the 
visible  color  region.  The  spectral  properties  in  the  IR  region  are  currently  undefined  and  provide 
a  topic  for  additional  research.  The  images  were  obtained  using  methods  developed  by  Vrhel  and 
Trussell  [3].  This  paper  introduced  three  distinct  methods  with  which  spectra  could  be  generated. 
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The  work  was  extended  by  this  effort  to  use  prototype  spectra  to  help  generate  spectra  that  have 
random  characteristics  but  are  statistically  similar  to  those  in  the  class  from  which  the  prototype 
comes.  The  results  from  this  work  include: 


1.  Implemented  spectral  generation  methods  developed  in  [3].  The  methods  used  were 

(a)  copulas,  based  on  probability  density  transformations, 

(b)  projection  onto  convex  sets  (POCS),  based  on  defining  sets  representing  statistical  prop¬ 
erties  of  the  class, 

(c)  neural  networks,  based  on  training  the  network  using  members  of  the  class. 

2.  Employed  the  methods  to  produce  random  spectra  that  may  be  representative  of  a  class  of 
object  in  the  image.  This  was  done  primarily  by  use  of  a  mean  spectra  that  is  representative 
of  a  class. 

3.  Evaluated  methods  with  respect  to  computational  efficiency  and  ease  of  use.  The  basic  results 
are  that  POCS  is  the  most  versatile  method  but  is  significantly  slower  than  the  others.  The 
Neural  networks  approach  had  significant  problems  in  attaining  the  flexibility  needed  to  easily 
represent  different  classes  of  objects. 

4.  Developed  methods  to  produce  textured  images  with  random  variations  that  are  perceptually 
uniform  across  the  image.  The  bounds  on  the  sets  used  in  POCS  can  be  varied  according  to 
the  region  of  the  perceptual  space  used  for  the  problem. 

5.  Created  a  small  database  of  classes  of  hyperspectral  data  that  can  be  used  to  characterize  ob¬ 
jects.  Currently,  the  classes  include  such  common  objects  as  grass,  leaves,  asphalt,  bark(tree 
trunks)  and  various  colored  flowers. 

6.  Created  small  hyperspectral  images  using  these  methods. 


One  example  of  this  effort  is  shown  in  Figure  1,  where  we  display  the  original  RGB  image  and 
seven  bands  of  the  hyperspectral  image  created  from  this.  Note  that  the  deep  red  band  at  700 
nrn  looks  quite  different  from  the  red  band  at  GOOnnr.  This  property  of  distinguishing  objects  by 
their  different  signatures  in  many  spectral  bands  makes  hyperspectral  images  most  useful  in  target 
detection  and  classification.  Creating  such  images  will  be  useful  in  the  academic  and  commercial 
environments.  Some  of  these  results  have  been  presented  in  conferences  [1] .  Additional  submissions 
are  being  prepared. 

This  project  is  motivated  by  the  potential  uses  of  hyperspectral  data  for  ATR  which  exists,  but  is 
hampered  by  the  fact  that  most  military  hyperspectral  data  is  classified.  Even  if  the  a  majority  of 
graduate  student  research  assistants  in  the  relevant  disciplines  were  not  US  citizens,  the  classified 
data  would  limit  testing  at  U.S.  universities.  In  addition  to  simulating  a  single  image,  this  research 
will  allow  us  to  create  a  series  of  images  of  similar  scenes  that  represent  the  same  physical  objects 
recorded  from  a  different  perspective  or  under  different  illumination.  A  final  by-product  comes 
from  the  fact  that  the  algorithms  developed  to  create  the  simulations  can  also  be  used  to  estimate 
an  underlying  hyperspectral  image. 
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Original  image  (300x300) 


(a) 

Hyperspectral  image  at  450nm 


(c) 

Hyperspectral  image  at  550nm 


(e) 

Hyperspectral  image  at  650  nm 


iff) 

Figure  1:  (a)  original  RGB  image,  (b-h)  images 

2  Approach 


Hyperspectral  image  at  400nm 


(b) 

Hyperspectral  image  at  500nm 


(d) 

Hyperspectral  image  at  600nm 


(/) 

Hyperspectral  image  at  700nm 


(h) 

r  bands  at  400nm  to  700nm  at  50  nm  increments. 


2.1  Use  of  Low-resolution  Spectral  Images 


We  assume  we  have  the  responses  of  the  sensors  through  the  Red,  Green,  and  Blue  filters  that  were 
used  to  construct  the  color  image.  Suppose,  just  for  example,  that  we  have  a  spectral  resolution  of 
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3nm,  and  we  know  the  response  of  each  filter  from  400-700  nrn.  Our  filters  may  then  be  represented 
by  a  vector  of  100  bands.  We  refer  to  those  filters  as  fr,  fg .  and  ft,-  Define  the  input  image  to  also 
be  a  100  element  vector,  denoted  i\  over  the  same  range,  and  the  filtering  process  may  be  written 
as  a  vector  products, 


tr¬ 

=  i\fr 

(1) 

ig 

=  i\fg 

(2) 

h 

=  i\fb 

(3) 

For  the  case  of  an  image  that  represents  an  intensity  distribution  in  the  visible  range,  we  can  assume 
a  scene  of  reflectances,  r(\)  that  is  illuminated  by  a  radiant  source  whose  energy  distribution  is 
defined  by  /(A).  The  intensity  image  is  obtained  by  representing  the  illiminant  as  a  diagonal  matrix, 
L,  and 

i  =  Lr  (4) 

where  we  have  written  the  reflectances  and  intensities  as  vectors.  The  equations  of  can  be  written 

t  =  FTLr  (5) 

where  the  filters  of  eq.(l)  are  the  columns  of  F.  For  the  estimation  problem,  for  a  single  pixel,  we 
have  three  equations  and  100  unknowns  to  find  i.  This  as  an  extremely  underdetermined  estimation 
problem,  and  can  be  addressed  effectively  only  if  we  employ  significant  a  priori  constraints.  These 
constraints  will  be  many  of  the  same  ones  used  for  the  generation  of  spectra  for  a  desired  class  of 
object.  Thus,  we  will  concentrate  here  on  the  constraints  and  how  they  are  employed  in  by  the 
generation  methods. 

2.2  Short  Review  of  Approaches 

In  our  first  work,  we  implemented  the  three  methods  suggested  by  [3]:  copulas,  projection  onto 
convex  sets(POCS)  and  neural  networks.  We  will  review  those  methods  here  and  evaluate  their 
suitability  for  modeling  classes  of  objects. 

2.2.1  Copulas 

The  approach  using  copulas  is  derived  from  a  common  method  of  generating  multivariate  random 
variables  [4].  The  copula  arises  from  Sklar’s  Theorem  [5],  which  proves  that  any  joint  probability 
distribution  can  be  written  in  the  form 

F(x i,..^xN)  =  C[Fi(xi),...,Fn(xn)]  ,  (6) 

where  C(u±,  ...,un)  is  a  joint  distribution  function  with  uniform  marginals  (that  is  rq  =  F)(xj )  is 
uniformly  distributed).  Assuming  the  derivatives  exist,  the  joint  density  function  is  given  by 

f(x i,  ...,xN)  =  /i(xi)  x  ...  x  fN(xN)  c[Fi(xi),  ...,FN(xN)]  ,  (7) 

where  c  =  and  fi(xi )  is  the  density  corresponding  to  the  distribution  function  Ft(xt). 
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The  function  C  is  called  a  copula.  The  function  serves  to  “couple”  the  marginals  Fi(xi)  into  the 
joint  distribution.  The  function,  in  essence,  introduces  correlation  between  the  variables.  The  most 
common  form  for  the  copula  uses  a  multivariate  Gaussian  distribution  function. 

In  this  case,  the  process  for  generating  the  vectors  is  straightforward.  To  begin,  a  set  of  zero-mean 
unity-variance  correlated  multivariate  Gaussian  distributed  vectors  are  generated.  These  vectors 
are  easily  created  using  standard  commercial  software,  e.g.,  MATLAB. 

In  the  next  step,  each  component  of  the  vector  is  transformed  to  a  uniform  distribution  using  the 
Gaussian  CDF.  In  other  words,  if  the  Gaussian  distributed  random  vector  is  denoted  by  v,  then 
the  uniformly  distributed  vector  is  given  by 

u  =  5erf  (^)  • 

where  the  operations  occur  on  each  of  the  vector  elements.  Note  that  the  elements  of  u  will  remain 
correlated  since  the  elements  of  v  were  correlated. 

Finally,  the  elements  of  each  of  the  uniformly  distributed  random  variables  u  are  mapped  using  the 
appropriate  inverse  of  the  CDF  that  you  desire  of  your  final  variable.  For  example,  if  it  is  desired  to 
have  the  marginal  CDF  at  wavelength  i  be  Fj(xi),  then  the  random  reflectance  value  at  wavelength 
i  is  given  by 

[r]i  =  FrU[uM.  (9) 

Again,  note  that  the  elements  of  r  will  be  correlated.  The  exact  characteristics  of  the  resulting 
correlation  is  an  open  research  question.  For  most  previous  uses  it  was  sufficient  to  have  any  simple 
correlation.  In  our  case,  the  correlation  is  used  to  enforce  smoothness  in  the  spectral  domain. 

Evaluation:  The  copula  method  is  easily  modified  to  produce  spectra  with  a  given  mean  distribu¬ 
tion.  As  mentioned  above,  the  method  produces  correlated  spectra  but  the  exact  correlation  of  the 
output  cannot  be  specified.  The  method  is  quite  fast  compared  to  the  POCS  below.  Unfortunately, 
the  copula  method  cannot  be  modified  to  produce  spectra  with  a  specific  visual  color  appearance, 
i.e. ,  tristimulus  or  chromaticity  value. 


2.2.2  POCS 


(8) 


Set  theoretic  methods  have  been  applied  to  color  science  problems  [6,  7]  and,  of  course,  for  problems 
in  image  and  signal  restoration  [2].  Here  we  used  this  powerful  method  to  constrain  the  vectors 
such  that  they  have  the  desired  properties.  We  assume  that  they  are  properly  bounded,  have  a 
specific  mean  and  correlation,  have  a  variation  distribution  similar  to  a  particular  data  set  and  are 
appropriately  smooth. 

The  method  of  successive  projections  onto  convex  sets  (POCS)  is  an  iterative  algorithm  to  find  a 
point  in  C  =  H^=1Ck-  For  POCS,  the  sets  must  be  closed  and  convex.  It  is  also  assumed  that  the 
sets  have  a  non-empty  intersection  that  contains  a  feasible  solution.  The  update  &k+i  at  iteration 
k  +  1  is  found  by  sequentially  projecting  onto  the  N  sets.  Let  Pc(a)  denote  the  projection  of  a 
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onto  the  set  Cc,  i.e. 


a  a  £  Ct 
a  a  ^  C( 


(10) 


Pc(a)  = 


where  a  satisfies 


min  d(a.  a)  subject  to  a  £  Cc, 

a 

for  some  metric  d(-,  •).  Then,  the  updated  estimate  a^+i  at  iteration  k  +  1  is  given  by 


(11) 


afc+i  =  Pn  (Pn- i  (•  •  •  P2  {Pi  (flfc))))  •  (12) 

The  method  is  guaranteed  to  converge.  The  constraints  previously  mentioned  can  be  used  to  define 
appropriate  sets.  In  addition,  a  common  set  defined  by 

C/  =  {i:||t-FTi||2<A/}  (13) 

is  used  to  enforce  the  condition  that  the  simulated  pixel  is  consistent  with  the  recorded  data.  The 
matrix  F  represents  the  sensor  responses  or  CIE  color  matching  functions.  The  bound,  A f,  allows 
for  the  possibility  that  there  is  some  noise  in  the  recording  of  the  low-resolution  data. 

One  significant  results  of  our  research  has  a  been  the  development  of  methods  to  vary  the  bounds 
of  the  color  sets,  Eq.(13),  according  to  the  region  of  the  color  space  in  which  we  are  working.  This 
allows  us  to  produce  variations  in  the  output  spectra  over  different  classes  that  are  perceptually 
similar.  An  example  of  this  effect  is  shown  in  Fig.  2.  This  extension  has  application  beyond  the 
hyperspectral  simulation  problem  by  allowing  fast  computation  of  perceptual  metrics. 

To  use  the  POCS  method,  we  generate  a  set  of  M  spectra  of  length  N  stored  in  a  matrix  R.  This 
matrix  should  be  contained  in  the  intersection  of  the  following  convex  sets: 

c bound  =  {  R  I  rmin;  <  rtj  <  rmaXi  i  =  0, ...,  {N  -  1)  j  =  0, ...,  (M  -  1)  }  ,  (14) 

where  rtj  is  the  element  in  the  zth  row  and  jth  column  of  R  and  [rmini  rmaxJ  define  the  reflectance 
range  at  wavelength  i, 

Cmean  =  {  R-  |  1 1 R.1  —  -Mf|||  <  8mean  }  (15) 

where  1  denotes  a  vector  of  of  all  ones.  This  set  is  used  to  allow  generation  of  different  classes  of 
objects  based  on  their  mean.  The  object  class  can  also  be  represented  a  covariance  structure.  This 
set  can  be  defined  by 


Ccov  =  {  R  |  ||RRt  -  M(Kr  +  ffT)||F  <  Scov}  (16) 

where  ||.||F  denotes  the  Frobenius  norm,  r  is  the  desired  mean  of  the  data,  and  Kr  is  the  desired 
covariance  matrix.  The  values  of  8mean  and  8cov  are  be  selected  based  upon  a  confidence  interval 
of  the  estimated  covariance  matrix. 

Evaluation:  The  POCS  method  is  easily  modified  to  produce  spectra  with  a  given  mean  distribu¬ 
tion  and  covariance.  The  POCS  method  can  easily  be  modified  to  produce  spectra  with  a  specific 
color  appearance  by  using  the  set  of  Eq.(13).  The  major  problem  with  POCS  is  its  slow  speed. 
However,  for  cases  where  images  need  to  be  produced  infrequently,  but  used  often  once  they  are 
available,  the  method  is  useful. 
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Original  Image 
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(o) 


Simulated  image  using  POCS  using  Delta  Error^  =  4.5 
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(b) 

Figure  2:  (a)  Original  RGB  image,  (b)  RGB  version  of  hyperspectral  image  that  is  based  on  the 
original  but  has  spectral  variations  added. 

2.2.3  Neural  Networks 


To  use  artificial  neural  networks  (ANNs)  to  simulate  the  reflectance  spectra  requires  appropriate 
design  of  the  network  and  proper  training.  In  this  case,  the  input  into  the  network  will  be  white  noise 
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vectors  and  the  output  should  be  simulated  reflectance  spectra.  The  structure  of  the  ANN  should 
be  designed  based  upon  a  priori  knowledge  of  the  function  that  the  network  is  to  approximate.  It  is 
known  that  a  2-layer  feed  forward  network  with  a  sigmoidal  nonlinearity  can  be  used  to  approximate 
any  function  with  a  finite  number  of  discontinuities,  arbitrarily  well,  given  sufficient  neurons  in  the 
hidden  layer  [8,  9,  10].  For  this  reason,  we  will  concentrate  on  this  architecture. 


Given  a  N  element  input  vector  e,  the  N  element  output  of  the  2-layer  feed-forward  ANN  with  S 
neurons  in  layer  one  is  expressed  as 


r  =  £(e)  =  [V$(We  +  b)  +  c]  , 


(17) 


where  <b(x)  =  [</>(.xi), ...,  (/>(xs)]T,  b  is  an  S  element  vector  of  bias  terms  for  the  hidden  neurons, 
W  is  an  S  x  N  matrix,  V  is  an  N  x  S  element  vector  of  bias  terms  for  the  output  neurons,  c  is  a 
N  element  vector,  </>  is  the  sigmoidal  function,  which  is  given  by 


4>(x) 


2 

1  +  exp(— 2  *  x ) 


(18) 


and  the  output  function  is  given  by  ’I'(x)  =  [ip(x i), ...,  tp(xm)]T , 


ip(x) 


1 

1  +  e~x' 


(19) 


which  insures  the  output  values  will  be  bounded  between  zero  and  one.  The  parameters  V,  c,  W, 
and  b  are  optimized  through  appropriate  training. 


Evaluation:  The  neural  networks  method  is  the  least  versatile  of  the  three.  We  modified  the 
method  by  using  radial  basis  functions  but  with  little  improvement.  The  network  would  need  to 
be  trained  for  each  class.  From  our  experiments,  the  number  of  training  samples  for  a  class  would 
be  almost  as  many  as  the  number  that  would  be  generated  after  the  network  is  trained.  However, 
if  we  envision  a  library  of  trained  networks  (or  library  of  network  parameters),  the  process  would 
be  quite  fast.  We  did  not  investigate  the  possibility  of  creating  such  a  library. 


3  Conclusion 


Our  work  produced  several  extensions  to  the  work  of  Vrhel  and  Trussell.  A  more  complete  eval¬ 
uation  of  the  method  introduced  in  that  work  was  done.  The  creation  of  hyperspectral  images 
that  are  realistic  pictorially  by  using  low-dinrensional  spectral  images  is  now  possible.  There  are 
additional  enhancements  that  are  worth  pursuing  and  will  be  investigated  in  later  work. 
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